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[57] ABSTRACT 

A compact convenient hand-held, accelerometer-based com- 
puter control device is utilized to control on-screen animated 
characters presented by a computer-driven display in which 
the movement, persona, style, arid- voice of the character is 
controlled. The control device detects accelerations which 
arc used to provide the on-screen character a particular 
persona, style or voice quality determined by the user. 
Character motion is determined by acceleration patterns 
based upon a learned language. The system requires only a 
series of easily learned hand movement patterns for corre- 
sponding character control. 

The style or emotional content of the movement is specified 
direcdy from gross accelerometer outputs without pattern 
matching. The outputs can also be used in connection with 
speech synthesis to provide voice quality attributes to 
audible information. Tlie system has particular relevance to 
control of on-screen characters in interactive theater and 
virtual reality systems. 

13 Claims, 11 Drawing Sheets 
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Fig. 3 
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CONTINUOUSLY VARIABLE CONTROL OF 
ANIMATED ON-SCREEN CHARACTERS 

HELD OF THE INVENTION 

This invention relates to computer control and more 
particularly to a hand-held device for controlling movement, 
speech and persona of an on-screen graphical object, icon, or 
character without providing frictional contact or mechanical 
coupling of the device with a fixed surface or structural 
element. Particularly, it relates to multi-level control of 
continuously variable attributes of the object, icon, or char- 
acter. 

BACKGROUND OF THE INVENTION 

As described in U.S. Pat. No. 5,347^06, issued to Tohei 
Nitta on Sep. 13, 1994, and incorporated herein by 
reference, an electronic meeting place can be provided in 
which conferees or participants can meet through control of 
on-screen characters, the movements of which are under 
control of the individuals. The system described in this 
patent utilizes cartoon characters or icons in order to avoid 
the problems with full-frame video teleconferencing. 
However, in the portrayal of one's self in terms of an 
on-screen character, it is necessary to provide this character 
with a persona, style, or mood specified by the individual 
participant. As described in this patent, the persona is 
controlled by switches on a keyboard or in a separate box so 
that fuU control can be provided over the representation of 
the individual participant. 

It will be appreciated that the utilization of separate 
switches and key strokes is a cumbersome way to control the 
persona of the on-screen character. While it might be pos- 
sible to use the traditional computer mouse to in some way 
control character persona or style, such a system is not easy 
to use and requires the individual to be at the computer 
terminal. It would not be possible when using the computer 
mouse to sit-back in one's chair or move around during 
teleconferencing. There is therefore a need for a more 
practical and simpler way to control the on-screen charac- 
ters. 

More particularly, as is well known, a mouse is utilized to 
control cursors represented on a computer screen in which 
the position of the cursor is controlled through the frictional 
contact of the mouse's roller balls with a fixed surface, 
usually in the form of a so-called mouse pad. While the 
utilization of a mouse for computer control is indeed an 
eflScient method for control of computer operations, the 
mouse has not been utilized to control the persona or style 
of icons or on-screen characters but rather has been used to 
move on-screen objects by clicking on them and dragging 
them to different locations, with the length of the move 
proportional to the movement of the mouse over the mouse 
pad. The traditional mouse is therefore incapable of con- 
trolling the persona of a cartoon or other character which is 
presented on-screen in a conferencing or meeting scenario to 
represent the individual communicating with others in an 
electronic meeting place or other environment. 

Specifically, in order to get a graphical object on a screen 
to move in a given direction with a mouse, one must click 
on the object and then move the mouse to drag it by 
specifying each location along the desired path. Thus a 
graphical object cannot be made to move in a desired 
direction without specifying each point along the way. As a 
result, simple movements of the graphical object cannot be 
made by merely directing the graphical object to go in a 
desired direction. Thus the mouse is inefficient both to 
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specify simple movements and to give a graphical object a 
desired persona. 

By way of further background as indicated in the above- 
mentioned patent, the movement of the individual characters 

5 representing a particular person can be controlled through 
data gloves, data suits, chair input devices and the like. 
These devices are in some sense cumbersome, and it is 
therefore desirable to provide a convenient and preferably 
hand-held device which controls not only the movement of 

jQ one's individual character, but also the mood or the persona 
which the individual wishes to impart to his on-screen 
character. 

As mentioned above, in this patent cartoon characters are 
prestored to dramatically reduce the bandwidth required for 

15 teleconferencing. During the teleconferencing scenario, the 
individuals participating m a conference need to be able to 
control how they are presented to the conferees. This is 
accomplished through the provision of the associated per- 
sona commands, in one embodiment by providing physical 

20 switches at each of the terminals for each of the participants 
in a distributed teleconferencing network. 

However, the utilization of individual switches, or even a 
mouse driving a cursor on a screen menu is a cumbersome 
way in which to control what other conferees see of the 

25 individual conferee. There is therefore a requirement for a 
hand-held device which can be free of contact with a table 
or other flat surface that can be conveniently waved around 
by an individual conferee to permit control of his character 
without the requirement of wearing a data suit or a data 

30 glove, without the requirement of detecting hand or body 
position through optical recognition techniques which are 
computer intensive and slow, and without any physical 
attachment to a fixed structure to sense hand or finger 
position. 

35 Again, by way of background, hand gestures have been 
utilized in commimication systems for the deaf. U.S. Pat. 
No. 5,047,952 describes an instrvuncnted glove to produce 
signals corresponding to hand configiuration utilizing strain 
sensors. A system for converting an image to an animated 

40 cartoon-like drawing through a so-called contour extractor is 
fllustrated in U.S. Pat. No. 4,546383. U.S. Pat. No. 4,884, 
972 utilizes a mouse to select words for animated characters, 
whereas U.S. Pat. No. 4,906,940 describes a system for 
extracting features and images for machine control. 

45 Moreover, U.S. Pat. No. 5,008,946 describes a system 
sensing the movement of one's pupil and one's mouth to 
control a car, whereas U.S. Pat. No. 4,414,537 shows a 
man-machine interface utilizing a data glove. U.S. Pat. No. 
4,567,610 illustrates a reference histogram for pattern 

50 recognition, whereas the following articles and thesis 
describe hand gesture recognition systems and adaptive 
interfaces relating to hand gestures: 
Tomoichi TakahaShi and Fumio Kishino, Hand Gesture 
Coding Based on Experiments Using a Hand Gesture Inter- 

55 face Device, SICCHI Bulletin, April 1991; Interpreting Sign 
Language, IEEE Computer Graphics & Applications, Janu- 
ary 1994, pps. 36-37; Jennifer A. Hall, The Human Interface 
in Three-Dimensional Computer Art Space, media Lab, 
Massachusetts Institute of Technology, MSVS Thesis, Oct. 

60 18, 1985; and, S. Sidney Pels, Building Adaptive Interfaces 
with Neural Networks: The Glove-Talk Pilot Study, Depart- 
ment of Computer Science, University of Toronto, Tech 
Report No. CRG-TR-90-1, February 1990. 
It will be seen that while there are various methods of 

65 hand gesture recognition, they rely on multiple hand ges- 
tures for computer control. As such, signing recognition is 
much too complicated for machine control, as the difference 
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in the varioiis hand signs are difiScult to recognize, and the tation when grasped by the hand. This gives the user the 

language associated with hand signing too complex for perception or tactile feedback as to how to move the device 

machine control. to achieve, for instance, the desired direction of movement. 

Note that in terms of arcade games, joy sticks, and the like As contrasted to a ball-shaped device or any device with 

control graphical objects through the use of strain gages and 5 point symmetry, the use of the egg gives the device a 

switches, with the strain being measured between a fixed perceived up orientation to make it natural to specify other 

structure and the joy stick. These arc simple systems which directions with hand movements. 

do not analyze strain gage waveforms nor combine wave- . * j u .a- * .u 1 1 

forms for graphical ob^ct control. Thus, there are no com- . ^ ^f^"^^^ embodmient the acce erauons are normal- 

parisoDS or correlations with templates that can be used to «° 1°^^ gravitauonal field vector both to provide a 

establish the existence of a given gesture or series of '° stationary reference for the ^alys^ of the accelerations 

gestures. As a result television game control devices gener- regardless of the onentation of the device withm the hand, 

ally do not provide a suflacient level of sophistication to and also to resolve any aliasing or 180 degree ambiguity, 

control the persona of graphical objects representing human making device orientation irrelevant, 

beings. In a further embodiment, and to eliminate false activation 

Furthermore, such systems do not provide for adjustments of the graphical object control system, a button is placed on 

in the attributes of character motion or control. When a the device, in the egg embodiment at the top of the egg, 

motion is performed, it is always performed in exactly the ^hich is actuatable by the thumb in order to start the 

same manner. On the other hand, persons have different acceleration detection process. Prior to thumb activation, all 

characteristics to their motions depending upon the person, accelerations are ignored, whereas during the depression of 

the emotional state of the person, or context of the moUon. 20 ^^^^ ^^^^^^ acceleration data is transmitted for decod- 

All of the gesturmg systems fail to provide for vanable ^ j^-^j ^^j^^j 

characteristics of personal motion. * . j . t. . • -j j • t. j l 1 j 

Interactive pro-ams permit the control of certain char- , ^^e appreciated that what is provided is a hand-held 

acters in relation to other characters provided by a program. ^Z^'^^.u"^ untetfiered state and 

Such interactive programs may include theatrical produc- 25 '^^^.^^ no physical contact with a fixed structure for he 

' ^ i * u * ™ 4U ♦ * 1 provision of computer control information. As such the 

tions or virtual reahty systems. For such systems, the total , . . , - ^ . i_ . t 

* 1 L 4 ' 1 J- • *• • 4U 4* -u ♦ subiect hand-held device is said to be non-contactual. 

control of the character, mcluding vanation m the attnbutes ^ i . . -.i.- .t. 1. j j • 

r ij -J 1- *• * t- r Rather, the device fits easilv within the hand, and m one 

of motion, would provide a more realistic representation of ^*»^^*» ^^^^ ^« ^1, . . .1. . . 

i_ *T-**- * J * •♦ u ♦* embodiment is not umbilically tied to the computer, but 

a character. Existing systems do not permit such operations. , . . . ^ t • 

^t- • * u • rather commumcates with the computer control system via 

Furthermore, with mteractive programs, speech synthesis 30 ^^^^^ i-Mmuiuiiiv,ai^a witu wji^yui.vi v^u ^ y 

.J ' . c • . \' • f' r .1. • wireless data transmission, 
provides a portion of the interaction. Variation of the voice 

quality is imperative to provide a realistic character as ^h^ convenience of such an ergonometrically designed 

situations change. Existing control systems do not permit computer control device makes it easy for the particqjants in 

variations in speech synthesis. a° electronic meeting place type conference or in interactive 

35 theater to control their associated characters. During a 

SUMMARY OF THE INVENTION conference or theatrical presentation, for instance, an indi- 

What is provided in one embodiment for the control of vidual depicted by his/her character to be seated at a table 

graphical objects is a hand-held device which is moved or may be made to rise from his/her seat and pace off, as if in 

waved around by the individual in a scries of motions or anger; or may be made to appear either bored or agitated, 

patterns. Eadi pattern or series of patterns results in control 40 completely at the discretion of the individual controlling the 

of the on-screen character. In one embodiment, an egg- representation of himsel£^erself . Thus the movement of the 

shaped device is provided with internal orthogonally- character and the persona of the character can be easily 

oriented acoelerometers, with the sensed accelerations being controlled with a pleasing hand-held device in which the 

converted via an analog-to-digital converter to signals which orientation of the device within one's hand is not critical to 

are transmitted to the computer. The accelerations provided 45 the ability to effecmate character control or graphical object 

by different hand gestures or movements may be identifiable control through a limited series of highly-defined hand 

through the utilization of templates which provide histo- gestures or movements. 

grams of the patterns to be recognized. Correlation of raw As part of the subject invention, a series of gestures may 

data to the templates provides for the generation of control be considered to be a language such that when the gestures 

signals for generating the motion control of the appropriate jq are performed in a predetermined sequence, the correlation 

on-screen graphical object. is made to a particular type of computer control. Thus while 

In such embodiments the system includes a training the system will be described in accordance with the control 

session in which an individual trains the system for the of graphical objects, computer control through a language of 

particular patterns he desires for the control of the graphical a series of sensed motions is within the scope of this 

object in question. After the training session, the accelera- 55 invention. 

tions are utilized to control the on-screen graphical object so Iq summary, an imconstrained, hand-held computer con- 
as to either move the graphical object or provide it with trol device, in one embodiment accelerometer-based, is 
specific types of actions. utilized to control graphical objects in a computer-driven 

In another embodiment once particular movement is display in which the movement, attributes, style, and per- 

selected for an on-screen icon or character, raw accelerom- 60 sona of the graphical object is controlled through movement 

eter outputs are used directly to control the intensity or style and resulting accelerations of the hand-held device, such that 

of the on-screen character. The style can be used to illusUate computer control is effectuated without the requirement of a 

a specific mood or type of persona. The accelerometer roller ball contacting a fixed surface to provide positional 

outputs can also be used for modifying synthetic speech in input. In another embodiment, motions are used to provide 

order to maintain the mood or persona of the character. ss voice quality to speech of a computer represented character. 

Because in one embodiment the hand-held device is in the In one embodiment a series of easily learned hand move- 
form of an egg, the egg-shaped device has a natural orien- ments or gestures provide for the corresponding graphical 
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object control in which one or more hand gestures or set of parameters to control the emotive content of an 

movements are detected serially as a language to obtain the on-screen character; 

appropriate movement or persona of the graphical object FIG. 10 is table listing actions in terms of movement 

presented on-screen. In order to make device movement versus style to provide control of a perambulating on-screen 

recognition independent of device orientation, in a preferred 5 character as to emotive style; and 

embodiment accelerations in the X and Y directions are FIG. 11 is a block diagram of one embodiment of the 

referenced to the local gravity vector, making it unnecessary subject invention in which accelerometer outputs are used 

to ascertain the orientation of the hand-manipulated device directly to Control the style of a specified aaion without 

in order to provide highly recognizable hand movement and curve matching. 

gesture recognition. In one embodiment, the device is egg- lO DETAILED DESCRIPTION 

shaped to provide a preferential orientation of the device in j^^^^^^ ^ ^ ^ ^ .^^j ^j^^^^^^^ ^^^^ 

the individuals hand, thereby to be able to specify a general ^^^^ ^^^^^^^ ^ computer generally indicated at 10 includes 

up direcuon for the device. In another embodiment, a z a monitor 12 having a screen 14 in which a cartoon character 

direction accelerometer mcreases the number of hand move- j^pict cd. In order to control the movement or persona 

ments or gestures that can be recognize. With a button or 15 of character 16, a hand-held device 20 generally in the form 

switch at the top of the egg, in a further embodiment, ^n egg is grasped by a hand 21 so that the egg, when 

providing an actuation signal for the start of the detection picked up from its holder 22 can be made to move generally 

process, thus to eliminate false activations. In a preferred t^e directions indicated by A, B, C, D, E, and F thereby 

embodiment the control device is wireless for ease of use, ^ effectuate control of character 16, in one embodiment, in 

whereas in another embodiment, the control device includes 20 corresponding directions 

accelerometere and a processor mounted at the lop of a ^ appreciated that while the subject invention wUl 

wand wiUi electncid connecttons at the base of the wand ^e described in termsof the aforementioned movements, the 

bemg hard wired to the computer. Also as part of the subject movements of device 20 may be made to control different 

mvention any computer^ven graphical object may be ^^^^^^ 33 y^^j ^^jec However for 

controlled such that the hand-held device provides the 25 ease of discussion, it will be appreciated that in the embodi- 

requisite generahzcd computer control, with visual feedback ^^^^ ^^^^^ device 20 when moved from its original rest 

being provided by the display. ^^^^ 34 in the direction of dotted arrow A results in 

BRIEF DESCRIPTION OF THE DRAWING movement of character 16 in the direction of dotted arrow A 

_ . . - I .L o !_• . w -HI. on screen 14. Thus when grasping device 20 and moving it 

These and Other features Of the Subject Invem^^^ 30 from a rest position towafdsX right, character 16 moves 

better understood m comiection with the Detailed Descrip- ^^^^^^ ^ predetermined speed and for a prede- 

uon taken in conjunction with the Drawings of which: termioed len^ of time. 

FIG 1 is a diagrammatic illustration of the subject egg- likewise when device 20 is moved to the left as indicated 

shapedcomputercontroldeviceillustraUngmovementofthe ^ ^^^^^^ ^^^^^^^^^ ^ ^^^^ ^^^^ 

device and corresponding moUon and persona control of the 35 backward as illustrated by dotted arrow B on screen 14. 

mdividual s on-screen character; - ■*u j - a 

^ / . . .„ In a like manner with device 20 moved from a rest 

HG. 2 is a block diagram ofthe subject system lUustratmg position in an upward direction as illustrated at C, then 

the detection of device accelerations, the conversion of tiie character 16 can be made to move away from the plane of 

analog accelerations to a digital format, the transmission of ^^^^^^ y^he^as for a movement of device 20 from a rest 

the acceleration inforaiation to a receiver and the decoding ^ position in a downward direction as illustrated by dotted 

of the information to provide graphical object control; ^^^^ ^ character may be made to move towards the 

FIG. 3 is a series of vector diagrams illustrating the screen, 

motions that provide graphical object control for the graphi- Qf course, if the device is made to move in a circular 

cal objea of FIG. 1; pattern as illustrated by dotted arrow E, the character, in one 

FIGS. 4A and 4B illustrate a language comprising a series embodiment, may be made to do a flip as illustrated by 

of device movements which either individually or serially dotted arrow E on screen 14. 

provide for graphical object control, in which in one y^^^^ j^e above relates to motions of the character, it is 

embodiment the language includes a movement to specify possible for instance to provide the character with an agi- 

direction, followed by a movement to specify an action and tated state by moving the device back and forth in the 

followed by a movement to specify an intensity, all for the direction of dotted arrow F in a shaking motion which, inter 

control of the graphical object. ^Ua, could make the character 16's hand to move back and 

HG. 5 is a block diagram ofthe subject system illustrating forth as illustrated by double-ended arrow F on screen 14. 

preprocessing functions; What will be appreciated is that movement of device 20 

FIG. 6 is a more general block diagram of the subject 55 f^e air results in a predetermined motion or characteristic 

system. ofthe corresponding on-screen character or graphical object, 

FIG. 7 is a series of diagrams describing the motion of the with there being no tethering of the device to the computer- 
computer control device to achieve the indicated on-screen driven display, and without contact of the device with a fixed 
action of a predetermined character or icon; structure or object such as a mousepad. 

FIGS. 8A-8J are combined acceleration waveforms eo As will be described in connection with FIG. 2, device 20 

derived from the raw accelerations from orthogonally ori- may be provided with accelerometers to sense the motion of 

ented accelerometers within the computer control device of device 20, with the outputs of the accelerometers being 

FIG. 1, with the waveforms corresponding to various analyzed to determine the accelerations of the device. In 

motions of the computer control device to effectuate the order to eliminate false actuation, device 20 is provided with 

indicated on-screen character control; 65 a switch 26 depressable by thumb 28 such that only those 

FIG. 9 is a graph of intensity of action versus valence for motions are detected which occur during the depression of 

specifying an entire range of human emotion to arrive at a the switch. 



08/08/2003, EAST Version: 1.04.0000 



6,072,467 



8 



10 



20 



While the subject inveation will be described in terms of 
accelerometers, it will be appreciated that any means for 
sensing the motion of device 20 is within the scope of this 
invention. Whatever motion detection system is utilized, the 
actuation thereof via a switch on the device eliminates false 
indications of motion regardless of cause. 

Referring now to FIG. 2, device 20 includes orthogonally 
oriented accelerometers 30 and 32 oriented respectively in 
the X and Y directions, the outputs of which are normalized 
to the local gravitational field vector G by unit 34 to make 
device 20 orientation independent. 

The raw accelerations, normalized to the local gravita- 
tional vector, are converted &om analog form to digital form 
by an analog-to-digital converter 36, with the output of 
converter 36 being transmitted via transmitter 38 and 
antenna 40 to antenna 42 and receiver 44. The output of 
receiver 44 is decoded at 46 to ascertain the particular 
motion which has occurred via comparison of the accelera- 
tions transmitted to receiver 44 with accelerations corre- 
sponding to templates 48 which have been prestored in a 
training phase. The matching of the sensed accelerations 
with the template accelerations results in a decoded signal 
being applied to a graphical object control unit 50 which 
provides drive signals to a screen driver 52 to control the 
movement of a character or graphical object on display 14 
as described in connection with FIG. 1. 

What is provided is a wireless non-contacmal system for 
controlUng the movement or persona of a graphical object in 
the form of a character through the free-air sensing of 
accelerations of a hand-held device. 

In one embodiment, device 20 is in the form of an egg 
which automatically establishes an orientation for the device 
within one*s hand. It will be appreciated that the movements 
allowed in the subject context are in a plane parallel to the 
plane of the screen, with the X and Y accelerations not only 
normalized to the local gravitational field but also normal- 
ized to a plane parallel to the screen. 

It will be appreciated that an additional accelerometer in 
a direction orthogonal to the first two directions, such as the 
z-axis, permits the generation of three acceleration vectors, 4q 
thereby to extend the number of hand motions which can be 
recognized by the system. 

Referring to FIG. 3, it will also be appreciated that the 
motions of device 20 of FIG. 1 which are utilized to control 
graphical object 16 are those indicated by the dotted arrows. 45 
In this case, six motions of device 20 of FIG. 1 are 
recognized as motions which can eflfectuate different control 
scenarios for the graphical object. 

Referring now to FIG. 4A, note that a different sequence 
or series of motions of device 20 can result in a language to 50 
control the on-screen character to execute complex actions. 
For instance, as illustrated in FIG. 4A, the on-screen graphi- 
cal object corresponding to a character can be made to sit, 
cry, and to do so at a loud intensity. In this case device 20 
is moved to the right to signify through the detection of its 55 
accelerations a sit control signal. The first movement of 
device 20 in general indicates a movement of the graphical 
object. Thereafter, device 20 may be moved downwardly to 
pound against a fixed surface 60, thereby setting off a 
number of undamped transients to provide signals to control 
the graphical object to cry. The intensity of the crying may 
further be controlled by the movement of device 20 in the 
shaking motion illustrated by double-ended arrow 62 to 
indicate that the action described in the second of the device 
20 motions is to be more intense, whatever that action is. 

A language is therefore developed in which a first motion 
of device 20 may be a direction for the graphical object to 
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move, whereas the second motion of device 20 may specify 
an action, in this case crying with tears coming down from 
the eyes of the on-screen character. Finally, a further 
detected motion may be utilized to control the intensity of 
the action, in this case, the crying. 

Such a language is also illustrated in FIG. 4B in which 
movement of device 20 up and down as illustrated by arrows 
64 and 66 causes a movement of the graphical object, 
whereas quick rotation of device 20 in a subsequent move in 
the direction of arrows 70 and 72 causes a walking action of 
the on-screen character. Finally the same motion of device 
20 as that illustrated at the bottom of FIG. 4A causes the 
intensity of the action, in this case walking, to be faster. 

What will be appreciated is that by a series of device 
movements, various actions and personas can be imparted to 
the oa-screen character. 

Referring now to FIG. 5 in more detail, accelerometers 30 
and 32 are conventional monolithic accelerometers with 
signal conditioning. These devices are Available from Ana- 
log Devices of Norwood, Mass. as model AD XL 50. Accel- 
erometers of this type have general application for airbag 
deployment in which accelerations produce output voltages 
from 0 to 5 volts, plus or minus 50 G. The frequency 
response of these accelerometers is DC to 1 kilohertz. It will 
be appreciated that with a sampling rate of 244 hertz, the 
Nyquist limit is 122 hertz. In order to provide a robust 
system, low pass filters 74 and 76 filter out responses above 
72 hertz which in general are undetectable by the subject 
system, with the filters also filtering out any 120 hertz noise 
due to 60 cycle AC power supplies. 

The outputs of filters 74 and 76 arc applied to analog-to- 
digital converters 78 and 80. In one embodiment these 
analog-to-digital converters are 8 bit, 0 to 5 volt digital 
sampUng devices in general available as part of a formatting 
microprocessor 82, manufactured by Motorola Inc. of 
Scottsdale, Ariz, as Model MG68 HCll. It is the purpose of 
the analog-to-digital converters to convert the analog out- 
puts of each of the accelerometers as filtered and as illus- 
trated at 84 and 86 respectively into a digital format corre- 
sponding to records. In one embodiment the records are 4 
bytes long, with 1 byte for each accelerometer, 1 byte 
reserved and 1 byte for time stamp and phase information for 
the beginning and end of a record. The output of micropro- 
cessor 82 is provided to a host computer 84 whidi provides 
a number of preprocessing functions as illustrated at 86. The 
first of the preprocessing functions is a gravity normalization 
function as illustrated at 88. In order to accomplish this, the 
data is subjected to high -pass filtering to separate out data 
above and bebw 0.1 hertz. The data bebw 0.1 hertz is 
deemed to be gravitational data, whereas the data above 0.1 
hertz is deemed to be user data. It is then possible to subtract 
out the gravitational component from the user data to 
provide data normahzed to the gravitational field. It will be 
appreciated that aU the data can be represented as an X, Y 
vector with the gravity information being in a given direc- 
tion and the user data in another direction. The combination 
of these two vectors yields an angle by which the user data 
may be rotated to move from device coordinates to real 
world coordinates specified by the gravitational vector. In 
this maimer, graphical object control can be made indepen- 
dent of device orientation within the hand. 

It will be appreciated with orthogonally-oriented acceler- 
ometers that the orientation of the device within one's hand 
is irrelevant. However, when a third accelerometer is utilized 
in a mutually-orthogonal direction to the directions of the 
first two above-mentioned accelerometers, then it may be 
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important to be able to establish, other than by gravity, an up 
direction for the device for specifying rotation about the 
device's longitudinal axis. 

In terms of the shape of the computer control device, an 
important reason for an egg shaped or non symmetrical 
device is to give the user a feedback as to what, in his 
perception, is an up or down direction or in fact any other 
predetermined direction. While the system described above 
does not require foreknowledge of device orientation, it is 
convenient for the user in effectuating the hand movements 
he desires to believe that direction makes a difference in 
terms of the device be is holding. In short, the asymmetrical 
egg gives the user a means of telling the computer a given 
direction. 

Preprocessor 86 also provides a time expansion function 
as illustrated at 90 in which all gestures are normalized to a 
time scale, for instance, to 2 seconds. Thus if the, gesture 
takes shorter than 2 seconds, its effective length is length- 
ened to 2 seconds for comparison purposes. In order to 
accomplish the time expansion, half second pauses of less 
than one half G are detected to indicate the beginning and 
end of a gesture, with the accelerations therebetween 
expanded to this normalized time slot. 

Preprocessor 86 prior to gravity normalization and time 
expansion performs a scaling function as illustrated at 92 to 
convert numbers from 0 to 255 to an actual acceleration in 
terms of G, The result is the conversion of accelerations to 
a negative 33 Gs to a positive 33 Gs. 

The result of the preprocessing is applied to a pattern 
recognition circuit 94 which forms part of the decoding unit 
46 of FIG. 1. The pattern recognition matches the prepro- 
cessed accelerations to templates of accelerations which 
have been previously stored in a training phase such that 
with appropriate pattern recognition the corresponding out- 
put commands are transmitted to graphical object control 
unit 50 of FIG. 1. 

In one embodiment pattern recognition in the subject 
invention utilizes the K Nearest Neighbor Algorithm, in 
which the data comes out as 128 numbers which are then 
compared with the distance between all the previously 
stored samples as calculated in a 128 degree space. Here the 
correlation is the Euclidean distance. For example, given 
that a data point includes 128 different samples correspond- 
ing to a single point in a 128 dimensional space, each of the 
previotisly recorded samples is represented in 128 dimen- 
sion space such that when a new point in this 128 dimension 
space arrives, this data is compared to all previously 
recorded gestures as points in this 128 dimension space. K 
is an arbitrary number picked to find a gesture corresponding 
to -that set of points which is closest to a previously recorded 
gesture. It has been found empirically that a good number 
for K is the square root of the total number of points 
available. Of the K points found, the gesture detected is that 
which most quantitatively is represented in the K nearest 
neighbors. 

The K Nearest Neighbor algorithm is used to perform the 
pattem recognition computation required to classify a new 
control device movement as matching a particular recorded 
gesture. A database or learn set of gestures is first created by 
repeating each individual gesture or class ten times and 
recording each instance as an element of the class. Each 
element is represented by a vector of 128 real numbers. 
When a new input gesture is to be classified, a Euclidean 
distance is calculated between it and every element of every 
class of the learn set. These distances are then sorted in 
ascending order. A K value is then calculated as the square 
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root of the size of the learn set as recommended by Devijer 
& Kittler, Pattern Recognition Theory and Applications, 
New York, Springer- Verlag, 1987, and whichever class is 
most numerous in the first K distance values is determined 

5 to be the matching gesture. 

The Euclidean distance between two 128 dimensional 
vectors, is defined as the square root of the sum of the 
squares of the difference of each value: 

square root ((Al-Bl)+(A2-B2f ++{A128-B128f) 

10 A simple algorithm for identifying pauses in movement is 
used to separate gestures for pattem recognition. A pause is 
defined as motion less than 5.0 m/(s'^s) for a minimum 
diu-ation of 0.4 seconds. A gesture consists of the motion 
between two pauses, and is limited to 2 seconds; Gestiures 

15 that take longer than 2 seconds to complete are ignored. 
More generally and referring now to FIG. 6, the subject 
process can be described as follows. From a more general 
point of view, accelerator 100 provides an analog signal 102 
which is low pass filtered and applied to a DAC/UART 104 

20 which is the aforementioned 6811 microprocessor to provide 
a serial output 106 coupled to the serial reading port 108 of 
the aforementioned host computer. The output of the serial 
port is low pass filtered to find gravity at HO, with the 
gravity being subtracted off via a high-pass filter function 

25 112 to reference the accelerations to the local gravitational 
field as illustrated 114. This data, having been sampled at 
244 hertz is resamplcd at 30 hertz via a moving average filter 
116, the output of which is recorded at 118 for future gesture 
recognition. 

30 The output of filter 116 is also applied to a unit 120 which 
seeks to normalize the gesture detection system by looking 
for pauses to separate the gesmres, in one embodiment by 
looking for one half second pauses of less than one half G. 
Thereafter the data is split into less than 2 second segments, 

35 that being the arbitrary length of analysis for gesture rec- 
ognition at 122. Time expansion is accomplished at 124 
through an interpolation and resampling normalized on the 
2 second interval. The result is a 12)3 data point vector with 
data sampled at 32 hertz times 2 sensors and 2 seconds. The 

40 output is applied to the aforementioned pattem recognition 
unit 94 to produce the output commands or command 
strings. 

Referring now to FIG. 7, various movements of the 
computer control device result in various prescribed 

45 on-screen motions for the character or icon. For instance, 
movement of hand-held device 20 quickly to the right and 
then to the left indicates to the system that a character right 
turn is required. A left turn is illustrated when device 20 is 
moved quickly to the left and then to the right. An increase 

50 in speed is illustrated as a quick movement of device 20 
upwards, followed by a downward movement, and a 
decrease in speed is indicated oppositely. Reset is defined by 
a square movement of device 20, right, up, left, down. Four 
quick movements of device 20 right and left represents a 

55 sideways shaking, while an vertical shaking is represented 
by foiu^ movements up and down. Other motion patterns of 
the device 20, such as a figure eight, sway and circle can also 
be defined to correspond to specific types of motion. 
FIG. 8 illustrates the corresponding combined accelera- 

60 tion waveforms corresponding to the motions illustrated in 
FIG. 7. During a learning phase, each of the motions would 
be performed. After each motion, the corresponding motion 
is entered into the computer control. Corresponding motions 
could be selected from a set of predefined motions, or can be 

65 provided as a set of changes to the character. The keyboard 
can be used to enter the motions, or a different type of 
pointing device can be used during the learning phase. 
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During the learning phase, each acceleration pattern is stored 
and associated with the corresponding entered motion. Dur- 
ing operation, the acceleration patterns arc matched with the 
stored patterns to determine the desired motion. The char- 
acter is then moved according to the determined motion. 

According to another embodiment of the present 
invention, the outputs of the accelerometers to provide style 
or emotion to the character movement.'^ErG7J9 illustrates use 
of two parameters to select emotional content of motion. A 
first parameter, shown on the X axis, represents the intensity 
or strength of the emotion; a second parameter, shown on the 
Y axis, illustrates the valence or type of emotion. Several 
emotions represented by the combination of intensity and 
valence are shown. Thus, a low intensity represents happi- 
ness with a good valence, and sadness with a bad valence. 
Similarly, a high intensity represents surprise with a good 
valence and anger or fear with a bad va lence. 31ie;;acceler^ 
ometer outpjit g^an'.be-used to represe qtthclvatu^ 
iltKtwo-par^metersJoj^ 

iithe_accekrometerioSputs:areicontinuouspany-variation^f_^ 
^thejernotions can be selected by proper motionnDf the'device 

The emotional attribute can be illustrated in certain char- 
acteristics of the movement, particularly speed and ampli- 
tude. For example, a sadness can be represented by move- 
ments which are slow and have small ampHtudes; surprise 
can be represented by fast movements of greater amplitude. 

Alternatively, rather than translating outputs into a set 
emotions, the accelerometer outputs can be used to select 
certain direct characteristics of the motion, such as the 
~ speed, amplitude, and orientation. FIG. 10 illustrates various 
types of parameters which might be adjusted for a walking 
character. Each of these values can be independently deter- 
mined by a raw output of the device 20. The timing of the 
outputs can be used to represent each of the various 
parameters, and the value of the outputs represent the degree 
of the parameters. Each of these parameters arc independent 
and effect the overall perception of the motion. To obtain 
faster movement, the step speed or the stride length can be 
increased, with different effects in the presentation of the 
motion. 

The right hand column of FIG. 10 represents typical 
descriptions of types of walking movements. Each of these 
types of movements would correspond to a different set of 
values for each of the parameters. Furthermore, due to the 
continuous nature of the accelerometer outputs, each move- 
ment can be somewhere between the various types of 
movements. 

FIG. 11 illustrates processing of the accelerometer outputs 
in connection with parameter selection. Elements 100-118 
correspond to the respective elements shown in FIG. 6. 
These elements provide the output of the accelerometer 
corrected for the gravitational element and filtered to remove 
high frequency components. The outputs are then provided 
to parameter selection unit 152. Since parameters may 
depend upon the type of action, the specified action 150 is 
also an input to the parameter selection imit 152. Preferably, 
as illustrated in FIGS. 4A and 4B, the order of motions can 
be used to select a type of motion and then the intensity or 
style of motion. The pattern recognition apparatus shown in 
FIG. 6 can operate until a desire motion is determined. If 
parameters are associated with the motion, then subsequent 
outputs of the device 20 can be used to select the parameters, 
as shown in FIG. 11. Each parameter may be selected in 
order according to successive movements of the device 20, 
or the outputs of the separate accelerometers in the device 
can be simultaneously processed for selecting multiple 
parameters. 
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Block 154 provides for scaling the sensor outputs to 
correspond to a parameter raiige. The speed and force of 
operation of the device 20 depends upon ^e user. Therefore, 
in order to account for differences of users, without loss of 

5 rangps of parameters, the sensor outputs must be scaled. The 
modified parameter values are provided to an interpreter unit 
156 which produced command strings to control the move- 
ment. The parameters which are used to control the motion 
of the character on the screen are modified to provide the 

10 desired style of motion based upon the command strings. 
Parameter control can also be used to provide stylistic 
operation of other elements of a character besides motion, 
most particularly with respect to speech. In another embodi- 
ment of the invention, the parameters which are controlled 

15 relate to attributes of speech, such as cadence, amplitude and 
fundamental frequency. For interactive processes, speech 
can provide a significant part of character control. The user 
may select what is to be said at a given time, or may simply 
control the manner in which it is said. For interactive theater, 

20 as in real theater, the fines are predetermined, but the actor 
presents the lines in a stylized manner. Speech synthesizers 
can be used to generate the words. By controlling the 
parameters of the speech, various styles, moods or emotions 
can be represented by the dialog, 

25 The speech styles can be controlled jointly with motion or 
independently. The selected parameters can relate to emo- 
tions (as illustrated in FIG. 9) which are then translated into 
both movement and speech attributes. This would permit the 
consistent depiction of the character. Alternatively, the suc- 

30 cessive operation of the device 20 can be used to determine 
the motion attributes and speech attributes at different times. 

Additionally, the present invention is not limited to use for 
manipulation of on-screen characters. The parameter selec- 
tion process can be used with a speech synthesizer indepen- 

35 dendy of a character in order to provide stylized speech in 
any context. The parameter control can also be used for 
selection of similar types of parameters in music generation 
to create a desired style. The parameter selection process can 
also be utilized in non-character motion, such as for com- 

40 puterized operation of robots and vehicles. Furthermore, an 
accelerometer device is not necessary to provide the outputs 
for parameter selection. Any type of output which permits 
continuous variation in sensor outputs can be used. 

Having now described a few embodiments of the 

45 invention, it should be apparent to those skilled in the art that 
the foregoing is merely illustrative and not limiting, having 
been presented by way of example only. Numerous modi- 
fications and other embodiments are within the scope of one 
of ordinary skill in the art and are contemplated as falHng 

50 within the scope of the invention as defined by the appended 
claims. 

What is claimed is: 

1. An apparatus for providing stylized operation of an 
on-screen computerized character, comprising: 
55 a housing moveable in free air; 

at least one accelerometer, disposed within the housing, 
for generating an output representing an acceleration of 
the housing; 

means for selecting a value of at least one style parameter 
relating to a characteristic of an operation of a character 
based upon the output; and 

means for controlling operation of the character based 
upon the selected style parameter; 
65 wherein said operation includes speech, said at least one 
style parameter is one of a cadence, amplitude, and a 
fundamental frequency of the speech, and wherein said 
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Operation controlling means generates predefined 
words in accordance with the at least one stylized 
parameter. 

2. An object control system, comprising: 
a housing movable in free air; 

a motion detector disposed within the housing and con- 
figured to detect motion caused by movement of the 
housing in the free air; and 

a transmitter configured to transmit signals representing 
the detected motion to control an object without regard 
to a relative position of the housing in the free air; 

wherein the movement of the housing is movement which 
causes the housing to strike a rigid surface; 

wherein the motion detector is further configured to detect 
the motion caused by the striking of the housing against 
the rigid surface; and 

wherein the detected motion caused by the striking of the 
housing against the rigid surface is undamped transient 
motion. 

3. A method for controlling an object, comprising the 
steps of: 

detecting motion caused by movement of a device in free 
air; and 

transmitting signals representing the detected motion to 

control an object without regard to a relative position of 

the device in the free air; 
wherein the movement of the device is movement which 

causes the device to strike a rigid surface; 
wherein the detected motion is undamped transient 

motion caused by the striking of the device against the 

rigid surface. 

4. An object control system, comprising: 
a housing moveable in free air; 

accelerometers disposed within the housing and config- 
ured to detect accelerations caused by movement of the 
housing; 

a decoder configured to determine the movement of the 
housing based on the detected accelerations; and 

an object controller configured to control a persona of the 
object in accordance with the determined movement of 
the housing. 

5. A control system according to claim 4, wherein: 
the object is a cartoon character; and 

the controlled persona is one of a human emotion and an 
intensity of a human emotion. 

6. An object control system, comprising: 
a housing moveable in free air; 

accelerometers disposed within the housing and config- 
ured to detect accelerations caused by movement of the 
housing; 

a decoder configured to determine the movement of the 
housing based on the detected accelerations; and 

an object controller configured to control a movement of 
the object in accordance with the determined move- 
ment of the housing; 

wherein the controlled movement has a direction different 
than a direction of the determined movement of the 
housing. 

7. An object control system, comprising: 
a housing moveable in free air; 

accelerometers disposed within the housing and config- 
ured to detect accelerations caused by movement of the 
housing 



10 



IS 



20 



25 



30 



40 



50 



55 



60 



a normalizer configured to normalize the detected accel- 
eration to one of a local gravitational field vector and a 
time scale; 

a decoder configured to determine the movement of the 
housing based on the normalized accelerations; and 

an object controller configured to control an object in 
accordance with the determined movement of the hous- 
ing. 

8. An object control system, comprising: 
a housing moveable in free air; 

accelerometers disposed within the housing and config- 
ured to detect accelerations caused by movement of the 
housing and to generate outputs corresponding thereto; 

a scaler configured to convert the generated outputs to 
actual accelerations; 

a decoder configured to determine the movement of the 
housing based on the actual accelerations; and 

an object controller configured to control an object in 
accordance with the determined movement of the hous- 
ing. 

9. An object control system, comprising: 
a hotising moveable in free air; 

accelerometers disposed within the housing and config- 
ured to detect accelerations caused by movement of the 
housing 

a decoder configured to match the detected accelerations 
with template accelerations to determine an order of a 
plurality of movements of the housing; and 

an object controller configured to control an object to 
perform a type of action in accordance with the deter- 
mined order of the plurality of the movements of the 
housing. 

10. An object control system, comprising: 
a housing moveable in free air; 

accelerometers disposed within the housing and config- 
ured to detect accelerations caused by movement of the 
housing; 

a decoder configured to determine the movement of the 
housing based on the detected accelerations; and 

an object controller configured to control the object in 
accordance with the determined movement of the hous- 
ing such that speech attributes of the object are con- 
trolled. 

11. A control system according to claim 10, wherein the 
controlled speech attributes include at least one of cadence, 
amplitude and fundamental frequency. 

12. A control system according to claim 10, wherein the 
controlled speech attributes determine one of content of 
speech and manner of speech. 

13. An object control system, comprising: 
a housing moveable in free air; 

accelerometers disposed within the housing and config- 
ured to detect accelerations caused by movement of the 
housing; 

a decoder configured to determine the movement of the 
housing based on the detected accelerations; 

a speech synthesizer configured to generate speech asso- 
ciated with the object; and 

an object controller configured to direct operation of the 
speech synthesizer to control the object in accordance 
with the determined movement of the housing such that 
attributes of the generated speech are controlled. 
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